Skip to content

gh-160308: Buffer punycode incremental codec input until the final call - #162349

Open
kircova wants to merge 1 commit into
python:mainfrom
kircova:gh-160308-punycode-incremental
Open

kircova wants to merge 1 commit into
python:mainfrom
kircova:gh-160308-punycode-incremental

Conversation

@kircova

@kircova kircova commented Nov 21, 2026 •

Copy link
Copy Markdown

The punycode incremental encoder and decoder ran the whole-string transform on every chunk, so anything fed in more than one piece came out wrong ('a' then 'b' gave b'a-b-'). This makes them buffer until final=True, using BufferedIncrementalEncoder / BufferedIncrementalDecoder as suggested in the issue.

Behaviour changes:

  • Text files opened for writing with encoding='punycode' now write nothing. TextIOWrapper never calls the encoder with final=True, so the buffer is never flushed. Before this, a single write() of a whole label came out right. idna already behaves the same way (it drops the last label).
  • The incremental decoder no longer accepts str. It raises TypeError, like the idna one. codecs.decode(str, 'punycode') is unchanged.
  • Calling encode() / decode() without final=True returns nothing, and several separate labels passed through codecs.iterencode() get joined into one.
  • Reading a punycode text stream only returns text once it hits EOF.

Given these, I wouldn't backport it.

Docs: the issue says nothing in the docs mentions this, but the codecs table does say "Stateful codecs are not supported" for punycode. I updated that row.

Tests: new incremental tests in PunycodeTest, and punycode is no longer skipped in the incremental parts of test_basics, test_basics_capi and test_decoder_state. Removing those gates re-indents three blocks, so the diff is easier to read with whitespace hidden. StreamReader / StreamWriter are untouched, so punycode stays in broken_unicode_with_stateful for the stream block and test_seek.

gh-160155 has three xfail examples pinned to this issue. Whichever of the two lands second needs to drop them.

Used Claude to help with this.

@python-cla-bot

python-cla-bot Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

@read-the-docs-community

Copy link
Copy Markdown

Documentation build overview

📚 cpython-previews | 🛠️ Build #36059294 | 📁 Comparing 7ee43e1 against main (dc222c8)

  🔍 Preview build  

2 files changed
± library/codecs.html
± whatsnew/changelog.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant